[SPARK-7224][SPARK-7306] mock repository generator for --packages tests without nio.Path by brkyvz · Pull Request #5892 · apache/spark

brkyvz · 2015-05-04T18:50:05Z

The previous PR for SPARK-7224 (#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's Files to handle directory creation, and etc...

The description from the previous PR:

This patch contains an IvyTestUtils file, which dynamically generates jars and pom files to test the --packages feature without having to rely on the internet, and Maven Central.

cc @pwendell

I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, --jars was also failing after we turned off the --packages test in SparkSubmitSuite. It may be related to the launch of SparkSubmit.

use Ivy format address comments v0.1 addressed comments v0.2 new fix not using nio.Path

AmplabJenkins · 2015-05-04T18:52:10Z

Merged build triggered.

AmplabJenkins · 2015-05-04T18:52:18Z

Merged build started.

SparkQA · 2015-05-04T18:52:57Z

Test build #31773 has started for PR 5892 at commit e632381.

AmplabJenkins · 2015-05-04T20:17:09Z

Merged build triggered.

AmplabJenkins · 2015-05-04T20:17:18Z

Merged build started.

SparkQA · 2015-05-04T20:17:52Z

Test build #31776 has started for PR 5892 at commit 68214e0.

SparkQA · 2015-05-04T20:34:07Z

Test build #31773 has finished for PR 5892 at commit e632381.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "public class " + className + extendsText + " implements java.io.Serializable

AmplabJenkins · 2015-05-04T20:34:12Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-04T20:34:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31773/
Test PASSed.

SparkQA · 2015-05-04T21:57:11Z

Test build #31776 has finished for PR 5892 at commit 68214e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "public class " + className + extendsText + " implements java.io.Serializable
- trait LDAOptimizer
- class EMLDAOptimizer extends LDAOptimizer
- class OnlineLDAOptimizer extends LDAOptimizer

AmplabJenkins · 2015-05-04T21:57:15Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-04T21:57:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31776/
Test PASSed.

AmplabJenkins · 2015-05-11T20:42:14Z

Merged build triggered.

AmplabJenkins · 2015-05-11T20:42:20Z

Merged build started.

brkyvz · 2015-05-11T20:43:59Z

ping @pwendell this should stabilize SparkSubmitUtilsSuite tests

SparkQA · 2015-05-11T20:44:06Z

Test build #32420 has started for PR 5892 at commit e9b1903.

SparkQA · 2015-05-11T22:27:47Z

Test build #32420 has finished for PR 5892 at commit e9b1903.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "public class " + className + extendsText + " implements java.io.Serializable

AmplabJenkins · 2015-05-11T22:27:52Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-11T22:27:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32420/
Test PASSed.

JoshRosen · 2015-05-22T22:30:38Z

Jenkins, retest this please.

JoshRosen · 2015-05-22T22:31:27Z

Hey @brkyvz, what's the status on this? If the delta from the previous reverted PR to this PR is really small, then it's probably okay to pull this in to re-enable that test.

AmplabJenkins · 2015-05-22T22:32:11Z

Merged build triggered.

AmplabJenkins · 2015-05-22T22:32:19Z

Merged build started.

SparkQA · 2015-05-22T22:33:14Z

Test build #33371 has started for PR 5892 at commit e9b1903.

brkyvz · 2015-05-22T23:02:10Z

@JoshRosen it is very small. The only difference is
import com.google.common.io.Files instead of
import java.nio.{Files, Path}

SparkQA · 2015-05-23T00:19:42Z

Test build #33371 has finished for PR 5892 at commit e9b1903.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "public class " + className + extendsText + " implements java.io.Serializable

AmplabJenkins · 2015-05-23T00:19:47Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-23T00:19:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33371/
Test PASSed.

JoshRosen · 2015-05-23T00:47:27Z

Ah, great. Going to merge this into master and branch-1.4 (1.4.0) so that we regain test coverage for this feature.

…sts without nio.Path The previous PR for SPARK-7224 (#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <brkyvz@gmail.com> Closes #5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator (cherry picked from commit 8014e1f) Signed-off-by: Josh Rosen <joshrosen@databricks.com>

…sts without nio.Path The previous PR for SPARK-7224 (apache#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <brkyvz@gmail.com> Closes apache#5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator

…sts without nio.Path The previous PR for SPARK-7224 (apache#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <brkyvz@gmail.com> Closes apache#5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator Conflicts: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala core/src/test/scala/org/apache/spark/deploy/SparkSubmitUtilsSuite.scala

…sts without nio.Path The previous PR for SPARK-7224 (apache#5790) broke JDK 6, because it used java.nio.Path, which was in jdk 7, and not in 6. This PR uses Guava's `Files` to handle directory creation, and etc... The description from the previous PR: > This patch contains an `IvyTestUtils` file, which dynamically generates jars and pom files to test the `--packages` feature without having to rely on the internet, and Maven Central. cc pwendell I also rand the flaky test about 20 times locally, it didn't fail a single time, but I think it may fail like once every 100 builds? I still haven't figured the cause yet, but the test before it, `--jars` was also failing after we turned off the `--packages` test in `SparkSubmitSuite`. It may be related to the launch of SparkSubmit. Author: Burak Yavuz <brkyvz@gmail.com> Closes apache#5892 from brkyvz/maven-utils and squashes the following commits: e9b1903 [Burak Yavuz] fix merge conflict 68214e0 [Burak Yavuz] remove ignore for test(neglect spark dependencies) e632381 [Burak Yavuz] fix ignore 9ef1408 [Burak Yavuz] re-enable --packages test 22eea62 [Burak Yavuz] Merge branch 'master' of github.com:apache/spark into maven-utils 05cd0de [Burak Yavuz] added mock repository generator

brkyvz added 4 commits May 4, 2015 11:13

added mock repository generator

05cd0de

use Ivy format address comments v0.1 addressed comments v0.2 new fix not using nio.Path

Merge branch 'master' of github.com:apache/spark into maven-utils

22eea62

re-enable --packages test

9ef1408

fix ignore

e632381

brkyvz changed the title ~~[SPARK-7224] mock repository generator for --packages tests without nio.Path~~ [SPARK-7224][SPARK-7306] mock repository generator for --packages tests without nio.Path May 4, 2015

remove ignore for test(neglect spark dependencies)

68214e0

fix merge conflict

e9b1903

asfgit closed this in 8014e1f May 23, 2015

Uh oh!

Conversation

brkyvz commented May 4, 2015

Uh oh!

AmplabJenkins commented May 4, 2015

Uh oh!

AmplabJenkins commented May 4, 2015

Uh oh!

SparkQA commented May 4, 2015

Uh oh!

AmplabJenkins commented May 4, 2015

Uh oh!

AmplabJenkins commented May 4, 2015

Uh oh!

SparkQA commented May 4, 2015

Uh oh!

SparkQA commented May 4, 2015

Uh oh!

AmplabJenkins commented May 4, 2015

Uh oh!

AmplabJenkins commented May 4, 2015

Uh oh!

SparkQA commented May 4, 2015

Uh oh!

AmplabJenkins commented May 4, 2015

Uh oh!

AmplabJenkins commented May 4, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

brkyvz commented May 11, 2015

Uh oh!

SparkQA commented May 11, 2015

Uh oh!

SparkQA commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

JoshRosen commented May 22, 2015

Uh oh!

JoshRosen commented May 22, 2015

Uh oh!

AmplabJenkins commented May 22, 2015

Uh oh!

AmplabJenkins commented May 22, 2015

Uh oh!

SparkQA commented May 22, 2015

Uh oh!

brkyvz commented May 22, 2015

Uh oh!

SparkQA commented May 23, 2015

Uh oh!

AmplabJenkins commented May 23, 2015

Uh oh!

AmplabJenkins commented May 23, 2015

Uh oh!

JoshRosen commented May 23, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants